Digitizing and Remediating Engineering Assessments: An Immersive and Transportable Faculty Development Workshop

The design and delivery of effective digitization for formative and summative assessments that are suitable for computer-based exam delivery remains an open problem across engineering curricula. Engineering-specific exam digitization challenges include the need to adequately evaluate design skills, solution structure, intermediate work, creativity, conceptual understanding, and in many cases, rigor that exceed the capabilities of rote multiple choice formats. In our high-enrollment engineering program, we have developed, implemented, and evaluated a six-week cross-disciplinary Assessment Digitization Innovation (ADI) Workshop that supports engineering faculty interest in developing computer-based examinations that are responsive to best practices. Authentic assessment formats and topics of focus include incremental solutions, multiple answers, design-by-selection, declarative statement formats, and question cloning practices. Further, the remediation of computer-based exams using digitized formats also enables new opportunities to enhance learner engagement, metacognition, and soft skills, which are highly amenable to faculty edification and are integral faculty development components of the ADI Workshop. The first ADI Workshop was conducted during the Summer 2016 semester. The experience included four face-to-face in-class sessions and two online modules. At the end of the 6-week program, each participating instructor showcased an online assessment that they had designed and developed as a result of the workshop. The topics of the pilot program included: 1) strategies to construct effective STEM assessments, 2) using relevant question types and features in Canvas, a learning management system (LMS), 3) implementing authentic assessment, 4) strategies to encourage academic integrity in online assessments, and 5) composing exemplar design vignette questions to reinforce connections between concepts to achieve integrative learning. The pilot cohort included 10 instructors and 16 Graduate Scholar Assistants (GSAs) currently teaching gateway Engineering and Computer Science courses at the University of Central Florida, and interacting with an estimated 6,200 undergraduate Engineering and Computer Science students. Upon conclusion of the program, anonymous feedback was collected from participating instructors, and was overwhelmingly positive. All respondents were “very satisfied” with the in-class sessions, the facilitators of the workshops, and the online modules. Specifically, they rated the program topics, examples, and resources provided to be highly relevant. The majority of the respondents agreed that the workshop will impact their future course design and development in beneficial ways, such as time-savings, convenience, student remediation, and the ability to serve large enrollments. 1.0 Introduction and Motivation In this paper, we describe the motivation, contents, and outcomes of the faculty development and training component of a transportable college-wide engineering digitization initiative at the University of Central Florida (UCF), referred to as the Assessment Digitization Innovation (ADI) Workshop. The ADI Workshop constitutes an important pillar of a viable digitized assessment ecosystem, which consists of digitization instructional pedagogies, engineering assessment design, and technology/personnel infrastructures/services for digitized assessment delivery. Together, these elements facilitate the successful digitization of suitable Engineering formative and summative assessments that are amenable to computer-based delivery. Instructional technologies that enable the reallocation of valuable instructional time and expertise towards the most beneficial high-gain teaching activities, while simultaneously minimizing lowgain logistical tasks, offer substantial gains to both the quality and the productivity of engineering instruction [1]. One computer-based instructional technology application possessing such potential is assessment digitization. Assessment digitization, which involves the computerbased delivery and auto-grading of formative and summative assessments, has achieved widespread adoption within certain disciplines outside of engineering. Innovations in assessment for STEM disciplines are urgently sought, especially given that student enrollment in undergraduate gateway courses at some institutions has increased considerably. The College of Engineering & Computer Science (CECS) at the University of Central Florida (UCF) undergraduate enrollment, for example, has increased by 30.2% from 6,535 in Fall 2012 to 8,507 in Fall 2016. The trend nationally is for undergraduate engineering foundation courses to enroll over one hundred students, yet these courses often possess limited faculty and graduate assistant resources. Meanwhile, the efficacy of homework assignments, lab reports, and reused exams continue to be undermined by open-resource search engines and Internet-based solution repositories. Computer-based testing offers specific solutions to these concerns. Such solutions include computer-generated formula-based question content for randomized and/or distinct content, rapid remediation, and in-person testing centers, which engage learners via physical attendance and high integrity delivery. Focusing on the quality of education, skills, and employability of our graduates in computingrelated fields, several faculty at UCF successfully piloted a cost-effective approach to achieve these goals, which was expanded and formalized for dissemination as the ADI Workshop presented herein. DeMara et al. describe the BLUESHIFT pedagogy which integrates computerbased evaluation with a close-knit review and learning cycle based on directed and open tutoring to collectively form an Evaluation and Proficiency Center (EPC) facility [2]. Pilot results were very encouraging, as students’ test scores indicated a 43% reduction in D or F grades compared to a section of the same course with the same instructor, using conventional delivery, and survey results included overwhelmingly positive responses from students regarding the effectiveness of pedagogical strategies (e.g., Exemplar Vignettes, content tutoring), assessment models (e.g., electronically delivered quizzes, flexible scheduling, use of testing center), and tutoring strategies (e.g., self-paced, exam results review). Readers are referred to [2] for content, benefits, and challenges of delivering digitized STEM assessments with the BLUESHIFT pedagogy. The hurdles to achieving effective digitization of engineering assessments can be organized into the three broad categories of content challenges, instructional challenges, and organizational challenges. Content challenges are typified by the need to administer creative design problems beyond rote multiple choice and the characteristic that lengthy engineering assessment often use mechanisms to confer partial credit. Thus, in the ADI Workshop, participating faculty are exposed to a palette of approaches to design engineering assessments with partial credit that are isomorphic to conventional pencil-and-paper based exams, but are deliverable electronically. They also practice applying them to their targeted course and are able to customize Score Clarification practices for the course, which utilize handwritten image files that are retained for strengthening the learner’s soft skills through one-on-one clarification with Content GTAs. Instructional challenges include the cold-start problem of building a viable test bank of digitized questions, whereas only selected topics in engineering fields have digitized test banks available from textbook publishers. Additionally, the efforts invested to digitize should have significant reuse potential to preclude reinventing the wheel for a longer-term payoff that can be pooled among alternate faculty teaching a course at their institution. Organizational challenges relate to change theory of the participants involved, both instructors and students alike. An additional organizational challenge is the need to provide a support services layer to deliver the assessments, ideally within a proctored testing facility, while maintaining the question bank via creation of technical updates and question “clones” to mitigate crosstalk among asynchronous test takers. The ADI Workshop addresses each of these challenges by engaging faculty in a sixweek development course via immersive experiences in the actual digitized environment that their learners will also utilize, as identified herein. 2.0 Literature and Related Works Although a comprehensive survey of the digitization of STEM assessments exceeds space available, we highlight selected works. Fellin and Medicus [3] describe the use of multiple choice assessments in geotechnical engineering, which elevated the performance level of undergraduate students via pre-test practice, and how students strongly prefer practice over theory in engineering content. The authors were convinced that digitized assessments used in this mode can impart long-term benefits and justify the effort required to construct such assessments. While studies have shown mixed reviews of teacher perceptions of online assessments, use of such assessments has been shown to correlate positively with overall course grades [4]. Further, while perceptions of online assessments have been mixed, many studies document the benefit of frequent online evaluation at the college-level. In one study of two sections of a statistics course, students given frequent pencil-and-paper memory tests averaged 86% on exams versus 78% in the control group (r = .44), where r denotes the coefficient of determination [5]. This testing effect is more easily leveraged when assessments are digitized and moved online. For example, Angus and Watson [6] administered online formative assessments to over 1500 business math students over the course of a semester, concluding that “regular testing undertaken with online methods enhances student learning,” particularly because of advantages unique to online delivery (e.g., randomized questions, instant feedback, multiple attempts). Challenges to authenticity of authorship and academic integrity of fully online courses is pervasive in the literature. In an attempt to deter cheating, Proctor Hub, ProctorU, ProctorFree and Remote Proctor NOW (RPNow) offer limited commercially-available products. However, continuously monitoring the student’s webcam feeds for eye tracking can be intractable, and test takers cannot be prevented from capturing questions using cameras in the background despite the requirement of a lockdown browser. Thus, either a dedicated testing center or a block scheduling of an existing computer lab are the approaches recommended in the ADI Workshop. Unfortunately, with the release of test questions, students may only learn the answer to the question rather than achieving learning outcomes. Thus, engagement of Test Proctors is recommended, as they can be useful in preventing exposure and authenticating submissions. Finally, in the ADI Workshop we extend the promising aspects of an “Open Tutoring Center” where tutors are available for targeted assistance [7] by resolving the challenge indicated by the authors about the absence of an effective, integrated, and verifiable assessment methodology. The instructional approach promulgated herein is presented in Section 3.2. Further rationale for the digitization of both formative and summative assessments in Engineering courses is supported by the testing effect [8], which implies that mastery learning can be enabled for complex concepts through frequent formative assessments supported with timely and thorough feedback. Affording students with increased opportunities for formative assessments allows them to utilize feedback for reflection and growth. Specific and structured feedback that can be provided via digitized formative assessments facilitate student mastery and have a positive correlation with student achievement. Further, digitized formative assessments allow instructors to mine assessment results for learning gaps and misconceptions, informing modifications to instructional approaches, pace, and ordering of content. Digitized formative assessments become a critical component of a comprehensive pedagogical framework, which maximizes the benefits of particular instructional strategies, while mitigating specific drawbacks. A comparison between some of the existing digitization approaches for STEM programs and the approaches covered in our ADI Workshop is provided in Table 2.1. Brigham Young University (BYU) [9] and University of Utah [10] Testing Centers (TCs) both provide online testing facilities for engineering and science programs that use digitized assessments. A commonality between these and ADI is the adoption of exclusive assessment delivery within a secured environment overseen by proctors. Students are provided with the increased convenience of being able to schedule appointments to avoid conflicts with their employment duties or exams in other courses. Schurmeier et al. [10] have studied the results of 10 years of digitized assessments on over 20,000 students using the University of Utah TC to address eight difficult topics in general chemistry. This effort provides a good example of potential benefits to the instructor derived from digitization of assessments, particularly the identification of trends in learners’ comprehension. Another significant digitization initiative is the Computer-Based Testing Facility (CBTF) at University of Illinois at Urbana-Champaign (UIUC) [11], which is being used for Table 2.1: Selected approaches for engineering digitization and their comparison to ADI Workshop. Approach Type STEM Programs Features BYU TC [9] Testing Center Physics and Astronomy Mechanical Engineering  exam problems are symbolic in nature which facilitates partial credit  equations not given in digitized assessments, but constants may be provided Univ. of Utah TC [10] Testing Center Chemistry  Item Response Theory used to identify difficult within topics general chemistry UIUC CBTF [11] Digitized Assessments Tools & Testing Center Computer Science Mechanical Engineering  assessment digitization includes interactive graphical response tool for STEM content  assessment delivery strategy allows unlimited retakes with highest score being retained ADI Workshop (described herein) Integrated Testing and Tutoring Methodology Civil, Computer, Electrical, Industrial, and Mechanical Engineering, Computer Science Information Technology  Palette of STEM question formats including Design-bySelection, Code Completion, Declarative Statement in Multiple Answer Format, Cloning Strategies, etc.  Utilizes scanned scratch sheets, score clarification services, review sessions. computer science and mechanical engineering courses. In addition to online testing and proctoring services, CBTF provides interactive graphical response tools for faculty to digitize their assessments. As listed in the last row of Table 2.1, the ADI Workshop promulgates an integrated testing and tutoring methodology, which thus far has been adapted to support a significantly broader range of STEM programs than previous approaches. Under the ADI Workshop approach, digitization enables auto-grading of assessments, which frees up graders for tutoring, a high-gain teaching and learning activity. Improving instructors’ methods and practices constitutes a critical role in any STEM reform effort. Brief, one-time interventions typically fail at changing instructors’ practices or attitudes toward innovative pedagogical approaches. We have chosen an extended professional development workshop environment because it allows more opportunities for instructors to address aspects of their teaching practice, explore various constraints in teaching and learning environments, and gain confidence to change instructional practices, which may engender persistent changes to instructors’ attitudes and behaviors [12]. While many studies have espoused the usefulness of professional development workshops in increasing instructor preparedness and effectiveness, STEM professional development workshops have not illustrated the same level of success in modifying instructor behavior as other content areas [13]. Thus, the ADI Workshop includes a flipped and blended classroom environment, active and learnercentered professional development activities, and extensive time and resources to reflect on existing pedagogical practices and making modifications to instructional approaches to implement “best practices” in STEM instruction. 3.0 Theoretical Framework While numerous efforts have been made to address the variety of problems currently facing STEM education, such as improving workforce development, increasing the number of women and underrepresented populations in STEM programs and careers, and implementing policies, supports, and processes to support enhanced STEM teaching and learning, many such efforts fail to be adopted [14]. Often times, this is due to the lack of design and development of a comprehensive change strategy prior to the implementation of the reform effort. Hence, it is critical to develop and employ a change strategy that extends the typical “best practice” approach that is typical in STEM reform efforts [15]. Rather, successful strategies of a comprehensive change approach should be continual, coordinated, and focused, and should address both changing the pedagogical conceptions of key and varied stakeholders in a STEM instructional system, as well as affording stakeholders with an iterative cycle of performance evaluation and continual feedback [16]. To address these issues, the ADI Workshop will develop and implement a comprehensive and expansive dissemination plan, derived from the Four Categories of Change Strategies model [17], and developed in an effort to provide environmental support to facilitate extensive adoption of processes and pedagogical practices associated with digitizing assessments. The change strategies to be employed will focus on: 1) developing extensive and expansive incentivized professional development opportunities for key faculty and administrators in an effort to alter instructional practices and policies, resources, and processes in place to support the integration of digitized assessments and associated pedagogical practices in STEM courses; 2) employing methods for aligning innovative instructional practices with the existing STEM conceptions of participating stakeholders, thus encouraging participants to explore innovative instructional approaches in a structured and measured manner; 3) developing an extensive and iterative plan for assessing and documenting the effectiveness of the ADI Workshop and accompanying pedagogical practices via stakeholder feedback; 4) disseminating program evaluation results to varied STEM programs and stakeholders; and 5) employing change agents to disseminate project materials and results. The aim of this comprehensive change strategy is to result in a wider adoption of the ADI Workshop content, processes, and strategies. Learners and faculty can benefit from shifting engineering gateway courses towards computerbased test delivery for those assessments that are suitable for digitization. Figure 3.1 depicts doing so by realigning educational and human resources without a net personnel increase using the BLUESHIFT pedagogy [2]. BLUESHIFT utilizes a taxonomy of online assessment instruments that facilitates design problems beyond rote multiple choice including multiple answer for partial credit, incremental solution multiple choice, and creative design via selection, which offer significant comprehension differentiation beyond the benefit of grading expediency. It also facilitates GSA-based open tutoring and remediation. A detailed financial cost model was developed whereas tutoring can be provided at no additional expense, by attaining a breakeven point between the grading hours avoided and the test proctoring hours required. This is shown to occur for a combined cohort of 1,150 students per term [2]. Thus, the BLUESHIFT pedagogy refocuses instructor and GTA roles from low-value repetitive tasks towards those having more significant impacts on learning outcomes, as depicted in Figure 3.1. Computerized testing centers support increasing enrollments while increasing practice and attainment of course outcomes, flexibility of scheduling, and rapid grading response. The approach set forth in the ADI Workshop is that the digitized assessment delivery infrastructure can be efficiently encapsulated within an Evaluation and Proficiency Center (EPC). As depicted in Figure 3.1, an EPC converts grading workloads into tutoring gains by re-mapping of graduate assistant expertise from low-impact grading tasks to new roles, which have increased impact on student outcomes. Thus, owing to time savings of autograding, the course graders are reallocated Figure 3.1: BLUESHIFT rebalances instructional expertise and classroom facilities towards the Learner’s frame of reference. Learners experience an increased frequency of higher quality online and live interactions; the moniker is analogous to Doppler blueshift in astronomy. Test Proctor Question Clone Composer Gradern GTA Grader1 ?